Skip to content

bench(reviewer-eval): weekly mining routine — local cron sweep, notify-only (sc-1415) - #321

Merged
norvalbv merged 2 commits into
mainfrom
bench/weekly-mining-routine
Aug 3, 2026
Merged

bench(reviewer-eval): weekly mining routine — local cron sweep, notify-only (sc-1415)#321
norvalbv merged 2 commits into
mainfrom
bench/weekly-mining-routine

Conversation

@norvalbv

@norvalbv norvalbv commented Aug 3, 2026

Copy link
Copy Markdown
Owner

Shortcut sc-1415 (epic 1399) — owner chose automation now over the wait-for-toil recommendation.

propose/weekly-mining.sh runs both miners (mine-bots, mine-telemetry) and both propose stages weekly, appending funnel summaries to ~/.claude-usage/weekly-mining.log and popping a macOS notification. Registered in the owner's crontab (Mon 09:00, already live), pointed at the main checkout so it always runs merged code — which now includes the GHSA miner's sibling scripts (#320); adding mine-ghsa to the weekly sweep is a one-line follow-up once its first batch validates the adapt flow.

Deliberately notify-only and local: mine-telemetry needs this machine's collector db + diff archive (a cloud routine could only cover the bot half), and adaptation stays a human/agent judgment session — the script never touches cases-*.jsonl and never commits. cron ships a minimal env, so PATH is set explicitly.

Runbook pending item 7 closed; decision note on benchmarks-grow-from-telemetry (2026-08-03). (Two earlier push failures were a corrupted worktree node_modules — same three tests failed deterministically there and passed everywhere else; clean reinstall fixed it.)

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added a weekly benchmarking routine that runs mining and proposal stages every Monday at 09:00.
    • Added notify-only logging and macOS notifications for routine results.
    • Fixture adaptation remains a manual step; no case files or commits are changed automatically.
  • Documentation

    • Documented the weekly routine, schedule, and manual adaptation workflow.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Warning

Review limit reached

@norvalbv, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 11 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: da7cea43-4ea9-41d2-a87f-61d7932bd641

📥 Commits

Reviewing files that changed from the base of the PR and between a645d83 and 1008610.

📒 Files selected for processing (3)
  • docs/benchmarks/corpus-growth.md
  • docs/decisions/benchmarks-grow-from-telemetry.md
  • gate-engine/review/eval/reviewers/propose/weekly-mining.sh
📝 Walkthrough

Walkthrough

A local Monday 09:00 cron workflow now runs mining and proposal stages, appends bounded output to a log, and sends a macOS notification. Documentation records the workflow and keeps fixture adaptation manual.

Changes

Weekly mining workflow

Layer / File(s) Summary
Mining workflow implementation and documentation
gate-engine/review/eval/reviewers/propose/weekly-mining.sh, docs/benchmarks/corpus-growth.md, docs/decisions/benchmarks-grow-from-telemetry.md
The script runs both miners and proposal stages with cron-compatible environment setup, bounded logging, and best-effort notification delivery. The documentation records the Monday schedule, dependencies, and manual adaptation boundary.

Estimated code review effort: 2 (Simple) | ~10 minutes

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the weekly local mining routine and its notify-only behavior.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 1
⚔️ Resolve merge conflicts 💡
  • Resolve merge conflict in branch bench/weekly-mining-routine
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch bench/weekly-mining-routine

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…y-only (sc-1415)

propose/weekly-mining.sh runs both miners (mine-bots, mine-telemetry) and both
propose stages weekly, appending funnel summaries to
~/.claude-usage/weekly-mining.log and popping a notification. Registered in
the owner's crontab (Mon 09:00), pointed at the main checkout so it always
runs merged code.

Deliberately notify-only and local: mine-telemetry needs this machine's
collector db + diff archive (a cloud routine could only cover the bot half),
and adaptation stays a human/agent judgment session — the script never
touches cases-*.jsonl and never commits. Runbook pending item 7 closed;
decision note recorded on the epic axis.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@gate-engine/review/eval/reviewers/propose/weekly-mining.sh`:
- Around line 22-32: Update the weekly mining script’s stage execution to
preserve each bun command’s exit status instead of relying on the pipeline
status from tail. Track failures for the miner and proposer stages, skip
propose/propose-telemetry.mts when mine-telemetry.mts reports no telemetry
candidates, and only show the “queues refreshed” notification when all required
stages succeed; propagate any failure as the script’s final exit status rather
than allowing osascript’s || true to mask it.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 15a45fb1-9e3c-43c3-96fc-9700e19ef74a

📥 Commits

Reviewing files that changed from the base of the PR and between 24bdbdd and a645d83.

📒 Files selected for processing (3)
  • docs/benchmarks/corpus-growth.md
  • docs/decisions/benchmarks-grow-from-telemetry.md
  • gate-engine/review/eval/reviewers/propose/weekly-mining.sh

Comment thread gate-engine/review/eval/reviewers/propose/weekly-mining.sh Outdated
@norvalbv
norvalbv force-pushed the bench/weekly-mining-routine branch from a645d83 to 3720c01 Compare August 3, 2026 13:57
…iew finding on #321

The stages ran as `bun ... | tail` pipelines whose status came from tail, so a
miner failure was masked, the notification still said "queues refreshed", and
the trailing `|| true` let cron record success. Each stage's own exit status
is now captured before its output is tailed into the log; a failure marks the
sweep failed, propose-telemetry is skipped when its miner failed, the
notification names the failed stages, and the script exits non-zero so cron
records it. (The same pipe-masks-status trap this session already hit with
backgrounded pushes — now fixed at both sites.)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@norvalbv
norvalbv merged commit ab0444c into main Aug 3, 2026
1 of 2 checks passed
norvalbv added a commit that referenced this pull request Sep 3, 2026
…ock exit propagation with a test (sc-2492) (#568)

Both logged sweeps (2026-08-24, 2026-08-31) failed every stage with nothing in the log but bun's "ulimit -n 2147483646 / launchctl limit maxfiles" hint. Two things were wrong, both in the runner.

- The 6-line tail cut off the error line printed just above that hint, so the cause of two failed weeks is unrecoverable. run_stage now keeps each stage's full output under ~/.claude-usage/weekly-mining/<sweep>.<unique>/<stage>.log (pruned by age after 8 weeks, never by count, so a concurrent sweep's directory is never touched) and, when a stage fails, logs its last 20 lines instead of 6. bun's root-error output is the last thing the process prints (error line, blank, 8-line hint block), so a 20-line failure tail always contains it; the full file is there for anything longer. No classifier decides what an "error line" is.
- Under cron the soft fd limit is 256 (launchctl maxfiles 256 unlimited), and bun's crash handler prints that same hint for ANY root error while the soft limit is under 16384 (src/crash_handler/lib.rs, the `Unexpected` branch), not only for EMFILE. A low limit therefore hides whatever the real error was. The runner lifts the soft limit to the hard limit (capped at 65536) before any miner runs, and the sweep header records the effective limits.

Is the exhaustion inherent? Every miner is synchronous (execFileSync only, no fan-out) and a cron-like replica (env -i, ulimit -n 256, the script's own PATH) ran mine-ghsa and mine-telemetry to exit 0; mine-bots was still sweeping 597 frink PRs at ship time and propose-bots depends on its output, so those two results are posted on sc-2492 rather than claimed here. No descriptor leak was found in the miners, so raising the limit is a diagnosability measure that also removes the one way a 256 limit could fail a legitimately larger sweep.

Exit propagation already existed (finding on #321); the regression test now locks it: a failing stub miner fails the sweep with exit 1, names the stage with its full-output path, puts the error line in the log, keeps the full output, and skips propose-telemetry; all-stages-failing names all four; two sweeps in the same minute get separate directories; a clean sweep exits 0. Stubs go through the script's own PATH ($HOME/.bun/bin first) with osascript stubbed so no notification fires.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant